A comparison of phone-based and on-site assessment of fidelity for assertive community treatment in Indiana.

OBJECTIVE This study investigated the reliability and validity of a phone-administered fidelity assessment instrument based on the Dartmouth Assertive Community Treatment Scale (DACTS). METHODS An experienced rater paired with a research assistant without fidelity assessment experience or a consultant familiar with the treatment site conducted phone-based assessments of 23 teams providing assertive community treatment in Indiana. Using the DACTS, consultants conducted on-site evaluations of the programs. RESULTS The pairs of phone raters revealed high levels of consistency [intraclass correlation coefficient (ICC)=.92] and consensus (mean absolute difference of .07). Phone and on-site assessment showed strong agreement (ICC=.87) and consensus (mean absolute difference of .07) and agreed within .1 scale point, or 2% of the scoring range, for 83% of sites and within .15 scale point for 91% of sites. Results were unaffected by the expertise level of the rater. CONCLUSIONS Phone-based assessment could help agencies monitor faithful implementation of evidence-based practices.